LGTM +1, Please resolve the conflict

Review: feat(core): support sort manifest entries by partition

Good motivation -- clustering manifest entries by partition will improve partition-pruning scan performance. Below are several issues I found while reviewing.

1. Loss of ADD filter during manifest reads (performance regression)

In the original readForFullCompaction, the code called:

manifestFile.read(file.fileName(), file.fileSize(), FileEntry.addFilter(), Filter.alwaysTrue())

Both mergeUnsorted() and mergeSortedByPartition() now call:

manifestFile.read(file.fileName(), file.fileSize())

...and then filter FileKind.DELETE in Java. This means DELETE entries are now deserialized from disk only to be discarded. For manifest files with many DELETE entries (e.g., after bulk deletes or partition drops), this is a regression. Please push the add-filter down into the reader call.

2. Parallel manifest reading was silently removed

The original code used sequentialBatchedExecute(reader, toBeMerged, manifestReadParallelism) to read manifest files in parallel. The refactored mergeUnsorted() and mergeSortedByPartition() now iterate sequentially over toBeMerged. For large tables with hundreds of manifest files needing compaction, this could significantly increase compaction latency. The manifestReadParallelism parameter is now effectively unused in the full compaction path.

The test testFullCompactionReadManifestsInParallel was modified to remove the parallelism assertions (maxConcurrentManifestReads), which confirms the feature was dropped rather than preserved.

3. IOManager created from System.getProperty("java.io.tmpdir")

In mergeSortedByPartition():

ioManager = IOManager.create(System.getProperty("java.io.tmpdir"));

This bypasses any user-configured temp directory or spill path. The commit code already has access to an IOManager (or its configuration) through the FileStore. Please pass the IOManager (or temp directories) from the caller rather than hard-coding the system temp directory. This also avoids creating a new IOManager per compaction.

4. Test assertions weakened to containsExactlyInAnyOrderElementsOf

Many tests (especially clustering/sort-related ones like testLocalSortClusterUnpartitionedTable) were changed from containsExactlyElementsOf to containsExactlyInAnyOrderElementsOf. This fundamentally weakens the test -- e.g., the comment in IncrementalClusterActionITCase says "verify internal order: within the single output file, rows must be sorted ascending by (a, b)" but then uses an assertion that does not verify ordering.

If the manifest sorting changes the read order for these tests, the correct fix is to add an ORDER BY clause to the query (as done in SparkWriteITCase) rather than dropping the ordering guarantee from the assertion. Otherwise these tests no longer validate that clustering produces correctly sorted data.

5. Inconsistent sort key between delta sort and full compaction sort

createManifestEntryComparator (used for delta sorting) orders by (partition, bucket, level, fileName), while mergeSortedByPartition (used for full compaction) orders by (partition, bucket, level). Without a stable tiebreaker in the full compaction sort, entries with the same (partition, bucket, level) have non-deterministic order. Consider adding fileName to the full compaction comparator as well for consistency and deterministic output.

6. Default values: manifest.delta.sorted=true changes write-path behavior silently

manifest.delta.sorted defaults to true, meaning every commit will now incur sorting overhead on the delta file list. Given that manifest.merge.sort-on-commit defaults to false explicitly to avoid commit-path latency impact, it seems inconsistent that delta sorting is on by default. Consider defaulting to false or at least calling this out in migration/upgrade notes since it changes existing behavior.

Minor

• The variable name partitionRmpr in createManifestEntrySortBuffer seems like a typo. Consider partitionCmpr or partitionComparator.
• The overload chain for merge() and tryFullCompaction() (3 overloads each) is getting long. A parameter object or builder might improve readability in the future.

Overall the feature is valuable and the use of BinaryExternalSortBuffer for spill-to-disk is a sound approach. Addressing the performance regressions (filter pushdown, parallel reads) and the weakened test assertions would strengthen this PR significantly.